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ABSTRACT 



This review examined the evaluation components from 10 
awarded state improvement grants designed to improve systems providing 
services for students with disabilities. The review examined the following 
aspects of the evaluation components: purposes, conceptual framework for the 
evaluation and systems reform, evaluation design and approaches, evaluation 
questions, types of data collected and analyzed, types of analyses conducted, 
and the evaluation of systems change and/or reform. The 10 states included in 
the review are: Alabama, California, Idaho, Iowa, Maryland, Massachusetts, 

New Hampshire, Ohio, Utah, and Virginia. The review notes that all 10 states 
addressed the 3 purposes for project evaluation specified in the request for 
proposals. The review also notes that different conceptual frameworks guided 
choice of evaluation designs and that evaluation designs influenced the 
questions formulated and types of data to be collected and analyzed. It 
reports that all 10 states will collect both quantitative and qualitative 
data concerning demographics, processes, and outcomes and that some states 
are collecting data at multiple levels including the individual level, school 
building level, district level, and state level. The bulk of the report is 
comprised of the individual state evaluation profiles. (Contains 14 
references.) (DB) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 




RESEARCH CORPORATION 



A Review of Evaluation 
Components from Ten 
Awarded State Improvement 
Grants (SIGs) 



by 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

M This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Beverly Mattson, Ph.D. 
and 

Ed McCaul, Ed.D. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



d- 

o- 



Oo 

0 



o 

rlEKIC 



RMC Research Corporation 
1815 N. Ft. Myer Drive 
Suite 800 

Arlington, VA 22209 

(703) 558-4800 (Main Number) 

(888) RMC-4200 (Toll Free) 

August, 1999 



BEST COPY AVAILABLE 

2 



Table of Contents 



Page 

Evaluation Guidelines in the RFP 1 

Review of Awarded SIGs and Their Evaluation Components 1 

Purposes of the SIG Evaluations 2 

Evaluators and/or Evaluation Partnerships 4 

Types and Focus of Evaluation Questions 4 

Evaluation Designs and Approaches 5 

Types of Data To Be Collected 7 

Types of Data Analyses To Be Conducted 10 

Evaluation of Systemic Change 10 

Examples of Four Approaches to the Evaluation of Systemic Change 11 

Summary . 15 

References 17 

State Profiles 

Alabama Evaluation Profile AL1 

California Evaluation Profile CA1 

Idaho Evaluation Profile ID1 

Iowa Evaluation Profile IA1 

Maryland Evaluation Profile MD1 

Massachusetts Evaluation Profile MAI 

New Hampshire Evaluation Profile NH1 

Ohio Evaluation Profile OH1 

Utah Evaluation Profile UT1 

Virginia Evaluation Profile VA1 




3 



Copyrighted by RMC Research Corporation, August, 1999 

A REVIEW OF EVALUATION COMPONENTS FROM TEN AWARDED 
STATE IMPROVEMENT GRANTS (SIGs) 
by Beverly Mattson and Ed McCaul 

EVALUATION GUIDELINES IN THE RFP 

The May 18, 1998 RFP in the Federal Register described the general guidelines for the 
evaluation section, Quality of Project Evaluation, of the State Improvement Grants (SIGs) 
applications. First, the application had to describe the extent to which the methods of evaluation 
were thorough, feasible, and appropriate to the project’s goals, objectives, outcomes, and the 
context. Second, the evaluation had to examine the effectiveness of project implementation 
strategies by using objective performance measures that were clearly related to the project’s 
intended outcomes and would produce quantitative and qualitative data. Third, the evaluation had 
to provide performance feedback and periodic assessment of progress toward achieving intended 
outcomes. 

Review of Ten Awarded SIGs and Their Evaluation Components 

As part of our interest in the State Improvement Grants and in ongoing work on evaluation 
of professional development and systemic reform, we reviewed the evaluation components of ten 
awarded State Improvement Grants. The intent of the review was to identify key concepts, 
strategies, and components and not examine strengths or weaknesses. 

We also drew upon the review presented by Dr. Carol Kochhar (1999), of George 
Washington University, at the 1999 CSPD Conference for Leadership and Change. According to 




1 



Copyrighted by RMC Research Corporation, August, 1999 



Dr. Kochhar, reviewers of SIG applications looked for a clear relationship between the evaluation 
conceptual framework and approach, the needs assessment, and the project’s implementation goals 
and objectives. There also needed to be a strong relationship between project goals and objectives 
and evaluation measures with clear, measurable benchmarks or outcomes. 

This review examines the following aspects of evaluation: (a) purposes; (b) a conceptual 
framework for the evaluation and systems reform; (c) evaluation design and approaches; (d) 
evaluation questions; (e) types of data collected and analyzed; (f) types of analyses conducted; and 
(g) the evaluation of systems change and/or reform. The ten states included in this review are: 
Alabama, California, Idaho, Iowa, Maryland, Massachusetts, New Hampshire, Ohio, Utah, and 
Virginia. See profiles on each state for more detailed information in the appendices. 

Purposes of the SIG Evaluations 

As described earlier, the SIG RFP clearly identified certain purposes for the evaluation 
section of the grant application, mainly performance feedback, periodic assessment of progress 
toward intended outcomes, and the effectiveness of the strategies implemented. All ten states in our 
review addressed these purposes. 

Other states included additional purposes. Table 1, Purposes of Evaluation Within Awarded 
State Improvement Grants, identifies all stated purposes in the ten awarded applications. Some 
additional evaluation purposes included: (a) the evaluation of the project’s effects on 
participants: (b) the assessment of the efficiency and effectiveness of the SIG design; (c) the 
examination of the impact of the SIG systems change model on professional development needs; 
(d) the provision of continuous information to leadership and stakeholders; and (f) 

ERIC 
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Table 1 

Purposes of Evaluations of Ten Awarded State Improvement Grants 
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Purposes of Evaluations 


Provide objective, efficient means of measuring and monitoring 
oroaress toward goals, objectives, initiatives, and outcomes 


Conduct internal evaluation to improve the manner in which the 
project activities are performed 


Conduct process evaluation of project to determine whether or not 
activities are being met in a timely and efficient manner 


Conduct formative evaluations to track the execution, usefulness, 
and outcomes of the project 


Provide a quantitative and qualitative evaluation that documents the 
project’s progress toward implementation and completion 


Ensure that the four initiatives accomplish their purpose 


Provide continuous improvement of the project 


Conduct a comprehensive evaluation of project implementation, 
milestones, and outcomes 


Evaluate the effectiveness of the project in meeting the grant 
purposes of the U.S. Office of Special Education Programs 


Evaluate the effect of the project on participants in grant activities 


Evaluate the project’s quantity , quality, and impact of services on 
intended audiences 


Evaluate the effectiveness of project’s activities in improving the 
state’s ability to meet the needs of children and youth with 
disabilities 


Evaluate project outcomes and their effectiveness 


Conduct summative evaluation to determine the project’s impact on 
accomplishing its missions 


— 1 

Evaluate the achievement of the project’s performance goals and 
objectives on a yearly basis and at the end of five years 


To assess the efficiency and effectiveness of the SIG design and 
content in meeting the proposed goals and objectives 


To examine the impact of the SIG systems change model on 
professional development needs 


To continuously inform leadership and relevant stakeholders of the 
status of the project 


To communicate the value of the project to relevant stakeholders 
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communication of the value of the project to stakeholders. These varying purposes 
influenced the evaluation design and approaches selected by the awarded states. 
Evaluators and/or Evaluation Partnerships 

Some states identified independent, external evaluators while others indicated that 
the evaluation would be conducted by an institution of higher education within their state. 
Some states referred specifically to evaluation partnerships. For example, Idaho 
specifically identifies an Evaluation Partnership (comprised of educators, university faculty, 
and family/community members). The Ohio SIG included an external evaluator, the 
Evaluation Services Center at the University of Cincinnati, an Evaluation Board, and 16 
regional evaluation centers located with State University Education Deans. (See Ohio 
Profile.) 

Types and Focus of Evaluation Questions 

According to Muraskin (no date), an effective evaluation provides information on the 
following questions: (a) To what extent are the purposes, goals, and objectives achieved? 
(b) What proportion of the target population is served? (c) What is the intensity of 
participation in services by those served? (d) What are the outcomes -- intended and 
unintended? (e) What is the evidence of effectiveness in meeting stated goals and 
objectives? (f) How can the program be improved? What “facilitates” success? 

Some SIGs formulated overall general process and outcome questions. Many 
process questions focused on the completion of activities and strategies. Outcome 
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questions most often focused on achievement of objectives, student outcomes, and system 
results. For example, Utah developed general overriding questions about the SIG’s three 
goals and specific questions for each goal and objective. Other states also developed 
evaluation questions matched to each goal and objective. 

The Virginia SIG describes three orientations -- learning, customer, and results-- 



t.hat shaped evaluation questions. For example 
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include: What factors in the project’s actions are influencing emerging results and in what 
ways? A customer orientation (Local Education Agencies, parents, other stakeholders) will 
be used to evaluate the project’s activities and services. A results orientation will focus on 
the achievement of planned results and unintended effects. 

Evaluation Design and Approaches 

The purposes and questions described above clearly shaped the evaluation design 
and approaches selected by states. An evaluation design refers to the purposes, 
questions, data collection and analyses, and variables included within a conceptual 
framework. The evaluation of a State Improvement Grant is a complex endeavor and 
decisions regarding evaluation design have political, social, and ethical dimensions. SIGs 
have multiple users and stakeholders, so it would not be surprising to find some 
cautiousness in describing specific designs and approaches. Perhaps because of the 
complexity, many SIGs provide only a general description of the types of evaluations they 
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will carry out and do not describe specific research designs or analytic techniques. 
Evaluation designs described in awarded SIG applications range from simple to complex. 

Most SIGs will be conducting formative and summative evaluations. Formative 
evaluation refers to the assessment of an innovative project to modify and improve the 
design to a particular context while it is still under development (Guba & Lincoln, 1981). 
Most SIGs are using formative evaluations to provide feedback and periodic assessment 
of progress toward outcomes. The purpose of a summative evaluation is to evaluate the 
completed project in terms of professional standards to certify its merit (Guba & Lincoln, 
1981). For example, California is using a yearly formative and summative evaluation 
process. Some States are using yearly and/or five year summative evaluations to evaluate 
whether project objectives have been met, to determine immediate outcomes, and 
determine the grant’s impact. New Hampshire and Ohio plan to conduct formative and 
summative evaluations at three levels to evaluate processes and outcomes. 

Other SIGs refer specifically to process, outcome, and impact evaluations. Process 
evaluations describe and assess processes, strategies, products, and activities. Outcome 
evaluations study the immediate or direct effects of programs and strategies on 
participants. For example, Alabama will be conducting process and outcome evaluations 
in a quasi-research design with pre- and post- interventions and matched control districts. 
Impact evaluations look beyond the immediate results of policies, strategies, and services 
to identify longer-term and unintended effects (Muraskin, nd). 
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Another model selected by Iowa and Massachusetts was the Discrepancy 
Evaluation Model (Yavosky, 1976). This model views program evaluation as consisting 
of a process of defining program standards, collecting performance information about 
these standards, analyzying and determining discrepancies, and using the discrepancies 
to change performance of program standards. 



New Hampshire will be conducting a multi-level evaluation of 



professional 



development based on Guskey’s (1998) and Kirkpatrick’s (1956) recommendations. (See 
the specific state profile for details). 

Idaho plans to use several evaluation approaches. First, the SIG will use 
Fetterman’s Empowerment Evaluation Model to build the system’s capacity and promote 
self-determination of participants. Second, the SIG will conduct process and formative 
evaluations to review the project’s activities and outcomes. Third, they will conduct 
outcome evaluations. 

Types of Data To Be Collected 

A variety of data collection methods will be used by the ten states. Some of the most 
common methods identified by the ten states included: needs assessments, surveys, 
questionnaires and ratings, interviews and computer-assisted interviews, student test 
scores on statewide and district-wide assessments, review of records (including student 
Individualized Education Programs, professional development plans, course transcripts, 
policies), and observations. 
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All ten states will collect a mixture of quantitative and qualitative data. Table 2, 
Types of Evaluation Data to Be Collected in Awarded State Improvement Grants, 
summarizes the types of data that will be collected by each state. The types of data can 
be categorized into three general types: (1) demographic data on participants, educators, 
parents and families, and students; (2) process data on partnerships, activities, strategies, 



materials, products, and World Wide Web site usage, and (3) outcome data for students, 



personnel, families, schools, local education agencies, and the system. We further 
subdivided some categories of data on Table 2 into subtopics (e.g., personnel data is 
broken down into personnel certification, preparation, recruitment, and retention). 

Some states also will collect these types of data at all levels: individual student and 
educator, classroom level, school-building level, local education agency, and state levels. 
See state profiles for particular details. 

Data for formative evaluations most often included: project activities, project 
strategies, professional development activities, etc. Data for summative evaluations most 
often addressed: project goals and objectives, project impact, data aligned with 
performance indicators, and outcome data. Many states delineated specific performance 
indicators for each goal and primary strategy. For example, Ohio identified performance 
indicators for students, parents, partnerships, collaboration, service delivery, and local 
outcomes. See profiles for specific examples. 
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Table 2 

Data To Be Collected in Awarded State Improvement Grants 



VA 


X 




X 


X 




X 


X 


X 


X 


X 


X 


X 




X 


X 




X 




14 


in 






X 


X 


X 




X 




X 


X 


X 


X 






X 


X 


X 




. BEST COPY AVAILABLE 


OHIO 


X 


X 


X 




X 




X 






X 










X 


X 


X 




HN 






X 


X 




X 


X 


X 




X 








X 


X 


X 




X 


MD 






X 


X 




X 


X 


X 




X 










X 


X 


X 


X 


MA 








X 


X 




X 




X 


X 




X 


X 






X 


X 




IOWA 












X 




X 




X 


X 


X 


X 


X 






X 






o 






X 






X 


X 


X 


X 


X 


X 


X 


X 




X 


X 




X 


CA 






X 






X 


X 


X 


X 


X 




X 


X 




X 


X 




X 


AL 


X 












X 




X 


X 


X 


X 


X 




X 








Types of Evaluation Data 


Partnerships/Collaboration 

Data on partnerships 


Data on agency collaboration 


Data on parent/family participation and/or 
satisfaction 


Data on Consumer Feedback 


Data on student lEPs and/or IFSPs 


Student Data 

Data on student participation & time in general 
education, LRE, and/or Vocational Education 


Data on student demographics & outcomes 


Data on student post-secondary education, 
employment, or community activities 


Personnel Data 

Data on personnel certification 


Data on personnel preparation (preservice and/or 
inservice) 


Data on personnel recruitment 


Data on personnel retention and Career Ladders 


Evaluation Data on training materials & other 
products 


Usage of World Wide Web Site 


Local Education Agency (LEA) or district data 


School-based data 


Preschool & early intervention data 


System Data 
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Types of Data Analyses to Be Conducted 

The analyses of data were briefly and generally described in the awarded grant 
applications. There were references to quantitative and qualitative analyses in general 
terms. Some states indicated they would conduct comparative analyses of specific data, 
Most states indicated that they would conduct disaggregation of statewide performance 
indicators, particularly student performance on statewide assessments. Some states 
indicated they would conduct trend analyses, particularly of student outcomes. New 
Hampshire referred to an analyses of inputs (e.g., professional development activities, 
demographics, and contextual variables) and student performance indicators. 
Evaluation of Systemic Change 

The evaluation of systemic change or reform is far more complex than simply 
evaluating the implementation and impact of a State Improvement Grant (SIG). For 
example, Heck (1999) suggests that the evaluation of systemic reform needs to identify: 
(a) districts, schools and other structures and functions within the system that will be 
involved; (b) the targeted components of reform efforts, and (c) the intended and 
unintended points of pressure and influence between the reform efforts and the system. 
Heck (1999) warns that maintaining a holistic perspective is important so that the big 
picture of reform is not lost in the details of implementation. As part of the evaluation 
process, states will need to: (1) understand and manage change through a large, complex 
system, (2) track the nature and extent of change over time, and (3) build and test their 
framework and conceptualization of systemic reform operating within their state’s context 
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(adapted from Heck, 1999). Rather than focusing on degrees of success of isolated 
components of reform, Banathy (1995) suggests that the evaluation process examine the 
totality and complexity of the reform operating within a system. Rather than evaluating 
discrete causes and effects, Julian, Jones, and Deyo (1995) suggest that the evaluation 
focus on the evidence that the reform effort as a whole contributed to successful solutions 
to entrenched problems throughout the system. 

Many awarded SIGS conceptualized a systemic reform framework and strategies. 
Some specifically described their evaluation approaches to systemic change and reform. 
Each state’s evaluation of systemic change will depend upon its conceptual approach and 
framework to systems change, the components that are being included and addressed, 
the strategies being used, and the outcomes anticipated and being evaluated. 
Examples of Four Approaches to the Evaluation of Systemic Reform 

We will briefly highlight the approaches taken by four states to evaluate systemic 
reform and/or change. Each state conceptualized systemic reform somewhat differently 
and selected different evaluation designs and approaches. The four states include Idaho, 
Maryland, Ohio, and Virginia 

Idaho. Idaho approached sustainable systemic improvement by building the local 
capacity to provide, improve, or expand services. The SIG identified the following 
components for effective systemic change: shared values and beliefs, state and federal 
legislation supporting ongoing efforts; united leadership and advocacy, collaborative 
structures at all levels, use of research and evaluation results to affect change efforts, build 

11 
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capacity for long-lasting change, and link with existing initiatives (Furney, Hasazi, & 
Destefano, 1997). The three primary strategies included changes in state and local 
policies and procedures, the implementation of School Improvement Plans under the 
Individuals with Disabilities Education Act (IDEA '97), and the incorporation of the 
Comprehensive School Reform Demonstration Program (CSRD). The SIG refers to a 
“synthetic policy framework” that will be used to understand the implementation of changes 
and improvements in systems. As part of the “synthetic framework” the Idaho SIG will draw 
upon concepts, insights, and propositions from previous implementation studies to develop 
new tools for analyses of policies. Comparative assessments for implementation 
experiences across policy areas or types of governmental units will be completed. The SIG 
will also use an Empowerment Evaluation Model (Fetterman, 1993) to build capacity and 
promote self-determination at all levels of the system. The evaluation design includes 
process evaluations and outcome evaluations. In addition, mini-research studies will be 
completed on status of programming for students involved in the project, changes in 

educational practices and personnel roles, and changes in parental participation. 

»• 

Maryland. Maryland conceptualizes the unification of systems through a 
comprehensive approach to school reform, the application of planned change, and shared 
participatory leadership and involvement across all levels. The Maryland SIG refers to 
Lockwood’s (1998) nine components for school reform including: support, measurable 
goals and benchmarks linked to state’s curriculum and standards, a comprehensive design 
with aligned components, effective research-based methods and strategies, aligned 
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professional development and personnel preparation, parental and community 
involvement, external technical support and assistance, coordination of resources, and 
evaluation strategies. Maryland will collect data on the. alignment of policies and 
procedures and linkages across education and other services. Additional data on proxy 
measures of systemic reform (e.g., development of and participation in project services) 
will be collected. Maryland will monitor changes in the capacities and practices of school, 
agencies, and institutions through the results of technical assistance agreements, annual 
surveys, outcomes of Local Education Agencies grants, and records of cooperation. 

Ohio. Ohio’s conceptualization of systemic reform refers to Sashkin and 
Ergemeier’s (1993) approach that focuses on fixing parts, people, schools, and the 
system. Ohio also refers to building a common vision and direction, decentralizing 
authority, and changing accountability. Ohio identifies their state’s model of Continuous 
Improvement Planning (CIP) as a primary systemic change strategy. Ohio also articulated 
anticipated systemic change outcomes. The anticipated outcomes included: (a) improved 
capacities to direct personnel development needs, (b) increased use of external 
professional development needs; (c) improved capacity to meet needs of consumers, (d) 
increased use of Ohio’s CIP model to meet the needs of students with disabilities, and (e) 
improved state capacity to meet the needs of children and youth with disabilities. Ohio’s 
evaluation design included a three tiered design with process and outcome components: 
Ohio identified agency collaboration indicators, service delivery indicators, and building- 
level process indicators. Regional coordinating councils will identify desired outcomes and 
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set benchmarks for local performance indicators, collect data, evaluate the extent to which 
goals, objectives, strategies and outcomes were achieved, and evaluate the impact of 
project activities on improving results for children with and without disabilities. The 
Evaluation Services Center at University of Cincinnati will coordinate local evaluation 
efforts and collect and analyze statewide performance indicators. In addition evaluation 
audits and self studies of project objectives and progress on performance indicators will 
be conducted by each local partnership site, Coordinating Council, and state team. An 
external evaluator will conduct focus groups with key evaluators and complete the 
evaluation audits. 

Virginia. The Virginia SIG refers to systems change efforts as including policies, 
incentives, cross-disciplinary training, technical assistance, and evaluation within school 
divisions, individual schools, and individual classes. The State Education Agency must 
create structures that lead to changes in behaviors at the teacher/student, parent/student, 
administrator/student levels for sustainability. The SIG refers to key factors in carrying out 
reforms and specific strategies for change (Anderson, 1993). The Virginia SIG will use a 
results orientation to evaluate the achievement of planned results and unintended effects 
and a learning orientation to evaluate factors influencing emerging results, final results, 
and unintended effects. The evaluation of the effectiveness of implementation will address: 
(a) results-based program improvement, (b) partnership infrastructures, (c) persons and 
families who work with children and youth with disabilities, (d) a coordinated system of 
technical assistance and professional development, and (e) the supply and diversity of 
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qualified personnel. A strategic evaluation will be completed to document the achievement 
of five year performance goals. 

Summary 

The evaluations of State Improvement Grants is a complex endeavor with political, 
social, and ethical dimensions. The purpose of this review was to identify the key 
evaluation components of ten awarded State Improvement Grants. The evaluation 
components and their descriptions of the ten awarded state applications ranged from 
simple to complex. The RFP identified three purposes for the evaluation: (a) examine the 
project’s effectiveness of strategies, (b) provide feedback, and (c) provide periodic 
assessment of progress toward the intended outcomes. All ten states addressed these 
purposes although a number of states included additional purposes. The conceptual 
framework for the overall purpose and evaluation of the SIG appeared to influence the type 
of evaluation design and approaches selected by states. For example, given the RFP 
purposes, most SIGs will be conducting formative and summative evaluations. In addition, 
several states selected other evaluation designs to complement- the formative and 
summative evaluations and address additional purposes. In turn, the evaluation purposes 
and design influenced the questions formulated and the types of data to be collected and 
analyzed. All ten states will collect a mixture of quantitative and qualitative data that can 
be categorized into three general types of data: demographics, processes, and outcomes. 
Some states will collect data at all levels: individually-focused, school building level, district 
level, and state level. 
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A number of states specifically referenced their evaluations to systemic reform and 
/or change which is more complex and challenging than simply evaluating the 
implementation and effectiveness of SIG strategies. Each state’s evaluation of systemic 
reform depended upon its conceptual framework to their system and its components, their 
approach to systemic reform and change, and the focus to be evaluated. 



We would like to acknowledge and thank Jackie Jones, Connecticut CSPD Coordinator, 
for her review and feedback of earlier versions of this analyses. 
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Alabama Evaluation Profile prepared by RMC Research Corporation, August, 1999 

Alabama State Improvement Grant 

Alabama State Department of Education Dates: 2/1/99 - 3/31/2003 
Special Education Services 
PO Box 302101 

Montgomery, AL 36130-2101 Project Director: Linda Smith 

334-242-8114 



Grant’s General Purpose: 

This improvement grant for the state of Alabama is a systems change effort 
aimed at improving the reading skills of students with disabilities, reducing the numbers 
of students with disabilities who drop out, reducing suspensions and expulsions for 
students with disabilities, and reducing special education teacher turnover and 
increasing the numbers of fully credentialed teachers. 

Conceptual Framework for Systems Change: 

Alabama approaches systems change through partnerships, and an on-line 
professional development academy, and mentorships. 

Purposes of the Evaluation 

(1) Provide an objective, efficient means of measuring progress and 
outcomes. 

(2) Provide Alabama with a quantitative and qualitative evaluation that 
documents the project’s progress toward implementation and completion. 

Evaluation Design and Approach 

Alabama’s SIG will conduct process and outcome evaluation in a quasi- 
experimental research design with pre- and post-intervention and non-interventions. 
The design will use pre- and post-test measures with matched control districts. 

The process evaluation will track the implementation of activities and the 
provision of training, technical assistance, and other project intervention activities by 
documenting their delivery. 

The outcome evaluation will measure the achievement of students with 
disabilities on the Stanford 9 Achievement Tests and the passage of Alabama’s Basic 
Skills Exit Exam; the suspension/expulsion rates of students with disabilities, dropout 
rates of students with disabilities, and student graduation rates. 

AL1 
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Type and Focus of Evaluation Questions 

Evaluation questions are clearly related to the project’s goals and objectives. 

Process Questions: The process evaluation questions address: (1) the 
implementation of project activities, (2) the identification of barriers, (3) interactions with 
subcontractors and stakeholders, and (4) the need, availability, and usage of resources. 

Outcome Questions: The outcome questions address: (1) achievement of goals 
and objectives; (2) impact of activities on goals and objectives; (3) use of new 
knowledge, skills, and plans, by participants who received training; and (4) observable 
benefits derived from the project’s implementation. 

Types of Data To Be Collected for Evaluation 

The Alabama SIG will collect the following data: 

Partnership Data: Numbers of university-school partnerships and the frequency 
and numbers of students tutored; the number of staff and hours involved in staff 
development. 

Student Outcome Data: achievement scores of students with disabilities using 
Stanford 9 Achievement Tests and the passage of Alabama’s Basic Skills Exit Exam; 
the suspension/expulsion rates of students with disabilities, the dropout rates of 
students with disabilities, tardy and truancy rates of students with disabilities, and 
graduation rates of students with disabilities. 

Training Materials: Number and type of training modules developed. 

Personnel Training Data: number of persons by position and parents trained, 
location of training, and type of training, and/or coaching. Training with Online Academy 
and the number of chat sessions with numbers/addresses of participants. Number of 
tuition awards provided for training leading to certification. 

Personnel Recruitment Data: Number of recruitment plans developed, initiation 
of legislative efforts, materials developed, collaboration strategies developed with 
universities, number of positions filled and open, and number of teachers certified. 

Personnel Retention Data: Numbers of coaches and new teachers coached. 
Hours of coaching per participant and number of low incidence coachings. 

Computer-Assisted Telephone Interviews (CATI) 
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Types of Analyses of Data to be Conducted 

The SIG grant will use a mixture of quantitative and qualitative analyses. Much of 
the data will be nominal so non-parametric statistical tests will be employed when 
appropriate. 
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California State Improvement Grant 

California Department of Education Dates: 2/1/99 - 1/31/2003 

Special Education Division 
PO Box 944272 

Sacramento, CA 94244-2720 Director: Alice D. Parker 

916-445-4613 



Grant’s General Purpose: 



This project will develop collaborative partnerships to improve services for 
children and youth with disabilities. The state improvement plan will serve as a guide to 
help system stakeholders concentrate improvement efforts and focus on future 
achievements and as a blueprint for those responsible for making system improvement 
by delineating and prioritizing training, technical assistance, and other systemic needs. 



Conceptual Framework for Systems Change: 

The California SIG approaches systems change through the use of partnerships 
and building the evaluation capacity and assessment expertise within lead agency. 



The California Special Education Division staff will serve as internal evaluators 
and Sonoma State University will provide consultation on evaluation design, 
instruments, and practices. 



Purposes of Evaluation 



(1) To evaluate the project’s quantity and quality of services and (2) evaluate the 
impact of these services on intended audiences. 



Evaluation Design and Approaches 

Methods used in the evaluation are intended to be “consumer friendly” - that is, 
progress on objectives is easy to evaluate and determine. The attempt was made to 
utilize existing sources of data; for example, data collected by the California Special 
Education Management and Information System was incorporated into evaluation. Also, 
information from the new quality assurance and focused monitoring system will be 
used. 



The California SIG will use both formative and summative evaluations: Formative 
evaluations : Two per year-after month 3 and month 9. (So information may be used 
during Partnership strategic planning session.) Arranged by Sonoma State University. 
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The focus of formative evaluation will address number, nature, and quality of project 
activities, approaches, and problems for continuous improvement. 

Summative evaluations: Summative evaluations will be conducted annually to 
determine the degree to which objectives have been met, including the quality of 
process and products and the significance of the project’s work after the award period. 

Types And Focus of Evaluation Questions 

Questions for the formative evaluations include: 



What is the number, nature and quality of project activities actually implemented by 
target date? 

What aspects need improvement and/or change? 

What approaches are working well and warrant expansion? 

What problems are anticipated in implementing the next phase and how might these 
be overcome? 



Question for the summative evaluations will include: 



To what degree have objectives been met? 

Types of Data to Be Collected For Evaluation 

The California SIG has a strong correspondence between goals, objectives, and 
evaluation measures since quantifiable measures are included in objectives. 

Student Outcome data: Performance of students with disabilities on statewide 
assessments (Stanford 9) and comparison of students with disabilities and their 
nondisabled peers. Rates of suspension and expulsion of students with disabilities. 
Percentages of students with disabilities earning high school diploma or GED. 

Student Postschool outcomes as defined by assessment instruments cited in the 
California standards. 

Parent Participation Data: parents participating in general education reform 
initiatives, parents on key committees, and parents on statewide committees for 
systemic reform. 
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Parent Satisfaction Data: Parent surveys on assistance/support for transition 
services. Percentages of parents rating transition services as “very satisfactory” or 
better. 

Personnel Certification Data: Rates of special education teachers on 30 day 
emergency or 30 day substitute emergency waivers; rates of special education 
teachers who are less than fully credentialed. 

Personnel Preparation Data: Percentages of special and general education 
teachers that indicate that inservice training, resources, and/or technical assistance 
have helped them implement research-based strategies in the classroom. 

System Indicator Data: Extent to which the needs of individuals with disabilities are 
addressed through (1) presence on key state-level committees, (2) response to 
technical assistance activities, and (3) increased participation in general education 
reform activities 

The amount of instructional time for students with disabilities spent in the 
general education classroom. (California Special Education Management and 
Information System has collected baseline information.) 

Types of Analyses Of Data To Be Conducted 

The one specific type of analyses described in the California SIG referenced 
the comparison of the performance of students with and without disabilities on state 
assessments. 
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Idaho State Improvement Grant 

Idaho State Department of Education Dates: 2/1/99 - 1/31/2003 

Special Education Section 
P.O. Box 83720 

Boise, ID 83720-0027 Project Director: Russ Hammond 

208-332-6919 



Grant’s General Purpose: 

The Improving Results Initiative for Idaho is based on a collaborative partnership 
designed to align and strengthen educational systems that contribute to high achievement 
for children and youth with disabilities. Partners in this project include the Idaho State 
Department of Education, local education agencies, Idaho Parents Unlimited, institutions 
of higher education, and others. 

Conceptual Framework for Systems Change: 

Idaho’s Improving Results Initiative addresses sustainable systemic improvement, 
and building the local capacity to provide, improve, or expand services. The SIG addresses 
changes in state and local policies and procedures, School Improvement Plans under 
IDEA '97, and incorporates the Comprehensive School Reform Demonstration Program 
(CSRD). The SIG refers to critical themes identified from a study of transition systemic 
change (Fumey, Hasazi, & Destefano, 1997). The SIG identifies links between the current 
initiative and other reform/restructuring efforts in the state. The SIG refers to a “synthetic 
policy” framework that draws upon concepts, insights, and propositions from previous 
implementation studies. Idaho will create systems improvement b 

Purposes of the Evaluation: 

The purpose is to conduct process and outcome evaluation of the project. The 
process evaluation will determine whether or not project activities are being implemented 
in a timely and efficient manner and that project outcomes are being met. 
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A second purpose is to answer the following outcome evaluation questions: 

(1) To what extent have project staff and partners delivered what they stated 
they would deliver? 

(2) How effectively have they delivered it? 

(3) To what level have project participants been involved in the implementation 
and evaluation of this project? 

Additional evaluation questions are generated for each evaluation area (project goal) under 
outcomes evaluation. 

Evaluation Design and Approach: 

Idaho Evaluation Partnership (comprised of educators, university faculty, 
family/community members) will meet yearly and use the Empowerment Evaluation model 
(Fetterman, 1993) to build capacity and promote self-determination. The Evaluation 
Partnership will identify, design, and carry out a matrix of evaluation activities to measure 
the processes and outcomes of the SIG. The Idaho SIG will also issue mini-research 
grants to graduate students and classroom teachers to strengthen the evaluation process. 

Components of the Empowerment Evaluation Model include: Training 
Evaluation - evaluators will teach others to conduct their own evaluation activities. 
Facilitation - coaches or facilitators helping others to conduct evaluations. Advocacy - 
Evaluators will make certain the project is meeting identified needs. Illumination - 
evaluator will point out ramifications and potential hidden meanings of data. Liberation - 
evaluator will help individuals take charge of their situation. 

The Idaho SIG evaluation design has two levels: 

Level 1 : Process evaluation that will focus on whether or not the project activities 
are being implemented in a timely and efficient manner and the project outcomes are being 
met. A formative evaluation will be used to review progress and compare project 
outcomes with those proposed. Idaho will use a Progress Evaluation Chart for each 
objective to track progress. The Idaho SIG will use nine sources of information to gather 
and monitor progress. 
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Level 2: Outcomes evaluation will answer the following questions: (1) To what 
extent have project staff and partners delivered what they said they would deliver? (2) How 
effectively have they delivered it? (3) To what level have project participants been involved 
in the implementation and evaluation of this project? 



Mini-Research Studies by graduate students and other participants will address 
student outcomes, comparison of schools, changes in attitudes of educators, changes in 
educators’ roles, retention of special education teacher, collaboration of faculty in 
Institutions of Higher Education (IHEs). 

Type and Focus of Evaluation Questions 

There is a strong correspondence between goals, objectives, and evaluation 
questions. 

Outcome evaluation questions: (1) To what extent have project staff and partners 
delivered what they stated they would deliver? (2) How effectively have they delivered it? 
(3) To what level have project participants been involved in the implementation and 
evaluation of this project? Additional evaluation questions are generated for each 
evaluation area (goal) under outcomes evaluation. Questions are framed in a yes/no 
format. 

Goal 1, local and state policies, questions on the adoption of standards, the 
participation of students with disabilities in statewide testing, the development and use of 
alternative assessments, local improvement plans, alignment of monitoring and school 
accreditation, the relationship between monitoring and accreditation teams, and the 
involvement of parents and stakeholders in accountability reviews. 

Goal 2, parent influence on reforms, questions on training in other languages, 
parent-training materials in alternative formats, parent participation in training reflected on 
child’s files, the development and implementation of training on school reform, the linkages 
between needs assessment and training, and parental data on training. 



Goal 3, professional standards, certification, and personnel development 
programs, questions addressing the realignment of programs and training activities with 
standards, training of paraprofessionals, accessibility of training in rural areas, seamless 
system of personnel preparation, joint training with parents, increased number of personnel 
trained, and level of retention. 
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Goal 4, linked system of preservice and inservice training, questions on IHE 
policies, use of incentives, partnership training for parents and school personnel, 
development of a training plan, collection of data to determine gaps in training, 
development of training resources, and the creation of a Training Clearinghouse. 

Goal 5, Adequate supply of qualified personnel, questions addressing 
conduction of projections, targeted areas for retention and recruitment, awareness 
campaign, minority recruitment, data on attrition, and creation of policies. 

Types of Data To Be Collected for Evaluation 

The Idaho SIG will collect the following quantifiable data from: artifacts and records; 
PATH action plans* policies, handbooks, and university course syllabi will be collected. 

In addition, Idaho SIG will collect quantitative and qualitative data from observations of 
participants implementing activities. Other data that will be collected include: 

Satisfaction with Project Data: Quantitative and qualitative data from surveys of 
partners, parents, representatives on committees, educators, arid training participants on 
satisfaction with project. 

Evaluation of Products data: Content, ease of use, usability, and lack of bias. 

Personnel data: Certification status, alternative certification rates, attrition rates* 
retention rates, employment of Idaho-trained personnel in Idaho, and numbers of 
individuals with disabilities in training and employed. 



Student data: Data on students with disabilities will include graduation rates, drop- 
out rates, participation in and performance on statewide and alternate assessments data; 
student academic progress, suspension and expulsion rates. 

Post-school Student Outcomes: Employment, wages above the minimum wage, 
participation in post-secondary education or vocational training, and access to community 
participation. 

Parent data: Quantity, quality, and effectiveness of involvement, participation in and 
evaluation of training, educational achievement of students whose parents were trained, 
and parent/educator relationships. 
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Training data: agendas, participant lists, percentages of students completing IHE 
coursework, evaluation of satisfaction with training, and certification data. In addition, the 
SIG will collect data on the employment of noncertified teachers, consultant specialists, 
and paraprofessionals. Professional development plans of districts will also be collected. 

School data: The SIG will collect the needs assessment data of participating 
schools (15 schools per year for a total of 75 schools). In addition, Idaho will collect data 
on the implementation of the Results Based Model school improvement plans (in 15 
schools per year for a total of 75 schools); improvement of student outcomes on Iowa Test 
of Basic Skills (ITBS)Tests of Academic Proficiency (TAP), Direct Writing Assessment 
(DWA)/Direct Mathematics Assessment(DMA); the number of students reading on grade 
ievei by grade 3 in every school; suspension/'expuision data of students with disabilities; 
school reform results; participation in technical assistance from the State Education 
Agency; the quantity and quality of parent involvement; reports from general education 
teachers on team process; and principal reports. 

In addition, the SIG will use statewide, district wide, and schoolwide data reports on 
each performance indicator to collect data, observations and observation record review, 
interviews, record reviews, and surveys. 

Types of Analyses of Data To Be Conducted 

A variety of analyses will be used by the Idaho SIG, including: (1 ) review of records 
and quantifiable data; (2) “analyses of all available data” on training needs; (3) quantitative 
and qualitative analyses of surveys and questionnaires by parents, educators, and follow- 
up surveys by participants; (4) comparative analyses of project outcomes with those 
proposed; (5) qualitative and quantitative analyses of observational data; (6) disaggregated 
analyses of student assessment data and percent of annual gains on statewide 
assessments by students; and (7) analyses of percent of gains in student performance on 
statewide assessments by pilot schools. 

Quarterly and yearly evaluation reports will be disseminated. 

Analyses of Systemic Change 

Idaho will use a “Synthetic Framework” to understand the implementation of 
changes and improvements in systems. Comparative assessment of implementation 
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experiences across policy areas or types of governmental units will be completed. Mini 
Research studies will be completed on; status of programming for students involved in 
project; changes in educational practices and personnel roles; and change in parent 
participation. 
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Iowa State Improvement Grant 

Iowa Department of Education Dates: 2/1/99 - 1/31/2003 

Bureau of Children, Family & 

Community Services 
Grimes State Office Building 

Des Moines, IA 50319-0146 Project Director: Lana Michelson 

515-271-3936 

Grant’s General Purpose: 

The purpose of the Iowa SIG is to address four issues regarding special education 
in Iowa: (1) over 70% of special education students have significant reading problems; (2) 
over one of every eight students is being taught by teachers not fully certified in the areas 
in which they are teaching, fewer than 100 are exposed to special education teachers who 
are persons of color, and virtually none are being taught by persons with disabilities; (3) 
the effectiveness of transitions that occur as students move from one educational setting 
to another and from an educational setting into the community needs to be improved; and 
(4) proficient, appropriately comprised teams for the development of Individualized 
Education Programs (lEPs) are needed, along with the assurance that all lEPs reflect high 
academic standards. 

Conceptual Framework for Systems Change: 

The Iowa SIG incorporates four initiatives into a single effort to improve the state’s 
systemic responses to the needs of students with disabilities. The SIG Literacy Initiative 
is part of the larger Iowa Literacy Initiative and focuses on improving the literacy skills of 
students with disabilities by improving the content knowledge and skills of their teachers. 
The SIG IEP Initiative focuses on improving academic performance of students with 
disabilities by improving the overall quality of lEPs developed and implemented by general 
and special educators. The SIG Endorsement initiative will address the Multicategorical 
Resource certification. The SIG Career Ladder Initiative will alleviate specific personnel 
shortages 

Purposes of the Evaluation: 

(1) Ensure that the four initiatives accomplish their purpose and work toward 
shared goals 

(2) Assess the effectiveness of the SIG Project 

(3) External Evaluation of SIG Literacy Initiative and Evaluation of SIG IEP 
Initiative 

(4) External Evaluation of SIG Endorsement Initiative and Evaluation of SIG 
Career ladder Initiative 
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Evaluation Design and Approach 

The Iowa SIG evaluation aligns with the Logic Model embedded within U.S. 
Department of Education’s response to Governmental Performance & Results Act (PL 1 0S- 
62). Formative evaluation of the project’s progress toward objectives will be based on the 
Discrepancy Evaluation Model (DEM) (Yavorsky, 1976) through monthly grant 
management by objectives. The DEM Evaluation cycle includes: planning, collecting 
performance information, analyzing and reporting discrepancies, resolving discrepancies, 
and building program designs. Summative evaluation will also be conducted at the end of 
each year for each of the indicators. 

Two levels of evaluation will be conducted: immediate outcomes related to 

program and services implementation and outcomes focused on improved results for 

educators, families, and students with special needs. Intermediate and end outcomes 
will be examined for each project objective. Product and process outcomes will be 
documented. 

Type and Focus of Evaluation Questions: 

To be determined by two external evaluators. 

Types of Data To Be Collected 

External evaluators will use a variety of methods that may include data from the 
Iowa Board of Educational Examiners, Iowa Department of Education, local education 
agencies, area education agencies, written and telephone surveys, face-to-face and 
telephone interviews, and site visits. Quantitative and qualitative data on intermediate 
outcomes and results will be collected for each objective. 

Strategy implementation data will be collected. 

Personnel Preparation Documentation: Documentation of training meetings 
(agendas, materials, dates, participants, etc., training provided). Multicategorical 
Preparation and Certification: transcripts, course selection, and progress of personnel 
in the endorsement process for Multicategorical Resource certification. 

Personnel and Personnel Certification Data: percent of multicategorical resource 
teachers; ethnicity/race of personnel; personnel representing persons with disabilities; 
certification status of personnel; and teacher performance data. 

Career Ladder Initiative Data: evaluation of Career Ladder Implementation; 
monitoring of participant progress and newly certified teachers. 
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Student Individualized Education Program (IEP) Data: percent of academic goals 
that are measurable, specific, and reflective of standards in general education; percent of 
lEPs that are high in quality and meet federal standards; percent of IEP Teams that 
include lEP-proficient general education teachers and administrators. 

Student Outcome Data: Accomplishment of annual IEP reading goals by students 
with disabilities, percent of students with disabilities who meet or exceed progress 
projected by IEP team, percent of students with disabilities with improvement in 
performance. 

Types of Analyses of Data To Be Conducted; 

SVSuiticategoricaS Preparation and Certification: Analysis of transcripts, course 
selection, and tracking progress of personnel in the endorsement process for 
Multicategorical Resource certification. 

Career Ladder Initiative: evaluate Career Ladder Implementation; monitor 
participant progress and newly certified teachers. 

Documentation of process and product outcomes. 

Results examined by documenting changes in student performance levels. 

Further analyses to be conducted by external evaluators. 
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Massachusetts State Improvement Grant 

Massachusetts State 
Dept.of Education 
350 Main Street 
Malden, MA 02148 



Dates: 1/1/99- 12/30/2003 

Project Director: Madeline Levine 
781-388-3300 



Grant’s General Purpose: 



The general purpose of the SIG is to provide statewide systems improvement to 
benefit all students. More specifically, the project seeks to provide quality pre-service 
and professional development activities for educators, parents, students, and other 
interested stakeholders. The SIG focuses on six focus areas: 



• Services in the natural environment during early intervention and preschool years. 

• Literacy skills instruction in the general education setting during kindergarten and 
elementary and school years. 

• Literacy skills instruction in the general education setting during middle school and 
high school years. 

• Transition for students 1 8-22 to postsecondary education and/or employment. 

• Restructuring of teacher preparation programs based on educational reform efforts. 

• Enhancing accountability systems across the full range of services for infants, 
toddlers, and students with disabilities. 



Conceptual Framework for Systems Change: 

The MA SIG strives to establish systems change through collaboration among a 
variety of stakeholders. “Participatory Action Research" (Whitney-Thomas, 1995), which 
emphasizes investment by partners, guides the project. Themes include parent 
collaboration, research to practice, interagency coordination, innovation, capacity 
building, and linkages to previously existing reform programs. 
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Purposes of the Evaluation 

The overall purpose of internal evaluation activities is to improve the manner in 
which staff and contractors perform project activities such as training activities, effective 
outreach activities, dissemination activities, etc. More specific purposes include: 

(1) progress in achieving grant objectives 

(2) effectiveness of the project in meeting OSEP’s purposes for the grant 
competition. 

(3) effect of the project on the participants in grant activities 

Evaluation Design and Approach 

For evaluation, Massachusetts will use the “discrepancy model" - The 
discrepancy model focuses on the discrepancy between what was intended to be 
accomplished and what was actually accomplished. 

Both formative and summative evaluation methods will be employed. 
Formative activities involve pre-and post-tests, satisfaction surveys for inservice 
training, follow-up services, material utilization, and activity logs. 

Type and Focus of Evaluation Questions 

An annual report will synthesize information and report on the project’s successl 
in meeting desired outcomes. For the first year, desired outcomes will include: 

Did the project start on time? 

Did the Project Advisory Committee meet as stated? 

Were materials developed as stated? 

Did training and professional development activities occur as planned? 

Did project participants have opportunities to comment on whether activities were 
well done and effective accomplishing goals? 

Has the project involved individuals from diverse cultures and roles? 

Have project staff effectively advertised the availability of training? 

Have staff provided follow-up to training activities? 

Are project materials available in accessible format? 
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Types of Data To Be Collected for Evaluation 

The Project Evaluation section of the grant identifies the following performance 
indicators identified as part of evaluation process: 

• Review of Project Management Logs 

• Review of Quarterly Reports by Project Director 

• Review and approval of project material by PAC 

• Participation rate for training activities 

• Satisfaction of participants in training activities 

• Review of sample of Individual Family Service Plans (IFSPs) on an annual basis. 

• Review of student outcomes (e.g., lEPs, assessment participation, state 
assessment scores for reading and English language arts, etc.) in sample of schools 
with staff or parents participating in project activities 

• Teacher/parent surveys 

• Review of participation rate and participant satisfaction for mentor programs. 

• Analysis of preservice coursework to include Education reform, curriculum 
standards, developing literacy, and teaching of diverse learners 

• Review of prospective educators results on the MA Teacher Test and tracking of 
passing rate for prospective educators in participant Teacher Preparation Programs 

• Review of cost data for pilot sites 

Also, training activities surveys, participant evaluations, and telephone interviews (as 
needed) will be employed. 

Types of Analyses To Be Conducted 

Types of analytic techniques are not described in detail. Advisory Committee will 
review evaluation information. No specific data analyses are described. 

Analyses of Systemic Change 

Systems-change analyses are incorporated into overall evaluation strategy (e.g., 
evaluation of effectiveness of partnerships, train-the-trainer model, etc.) . 
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Maryland State Improvement Grant 

Maryland Department of Education Dates: 2/1/99 - 1/31/2003 

Division of Special Education 
200 West Baltimore Street 

Baltimore, MD 21201 Project Director: Lucy Hession 

410-767-0242 



Grant’s General Purpose: 

“The plans for the Maryland State Improvement Grant are based on a vision of a 
unified system that centers around a core set of student outcomes and standards that 
define goals for the system, and there is a means for assessing student progress toward 
those outcomes. Curriculum and instruction are aligned with the standards and outcomes, 
and the entire core is supported by policies and programs that guide professional 
development and preparation, technical assistance for school improvement, and the 
allocation of resources. The unified system accommodates and supports diverse learners 
without unnecessary categorization of students or program resources. The system values 
flexibility and collaboration at all levels in order to promote student attainment of goals. To 
make this vision a reality, the mission of this SIG is to improve education and outcomes 
of students with disabilities through services and strategies that involve stakeholders and 
system components in sustained efforts to bring about progressive and positive change in 
services for students ages three to 21 ; in professional development across the spectrum 
of professionals, paraprofessionals, and parents; and in preservice preparation of general 
educators, special educators, and related personnel.” 

Conceptual Framework for Systems Change 

Maryland conceptualizes the unification of systems through: (a) a comprehensive 
approach to school reform, (b) application of principles of planned change, and (c) shared 
participatory leadership and involvement across all levels. The Maryland SIG refers to 
Lockwood’s (1998) nine components for comprehensive reform. Lockwood’s components 
include: support, measurable goals and benchmarks linked to state’s curriculum and 
standards, comprehensive design with aligned components, effective research-based 
methods and strategies, aligned professional development and personnel preparation, 
parental and community involvement, external technical support and assistance, 
coordination of resources, and evaluation strategies. 
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Purposes of the Evaluation 

The Maryland SIG will conduct formative and summative evaluations. The purpose 
of the formative evaluation is to track the assessment of the execution, usefulness, and 
outcomes of the grant. The purpose of the summative evaluation is to determine the impact 
of the project on accomplishing its mission of improving education and outcomes of 
students with disabilities. 



Evaluation Design and Approach 



Maryland will use formative and summative evaluations according to Fullan’s 
(1996) mode! for evaluating systemic reform and a simplified version of American 
Management Association’s (AMA) model on monitoring and evaluating services and 
activities. The AMA Model will use specified objectives, outcomes indicators, and timelines 
for each goal with management by an objective approach and input-process-output 
tracking 



The formative evaluation will track and assess the execution, usefulness, and 
outcomes of: the: (1) alignment of policies and procedures, linkages, and cross-sectional 
stakeholder involvement; and (2) proxy measures of systemic reform that include student 
performance data, program quality, development & delivery of professional development 
and its participation; participation and alignment in preservice preparation, early 
intervention systemic evaluation, capacity building for improving education and outcomes, 
and feedback from participants. 

A Consumer Review Group will evaluate information services and products on 
quantity, management of information, production, quality control, distribution, consumer 
satisfaction, and feedback. 



The summative evaluation will determine the impact of the project on 
accomplishing its mission of improving education and outcomes of students with 
disabilities. The summative evaluation will: (1) address student’s learning and progress; 
(2) monitor changes in capacities of schools, agencies, and institutions; (3) address 
practitioners’ understandings, uses of new practices to improve instruction, and 
practitioner’s needs and attitudes; and (4) address early intervention outcomes and 
indicators. 



In addition, the Maryland SIG will conduct an evaluation study of the quality and 
flexibility of local services to families with eligible infants and toddlers. 
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Type and Focus of Evaluation Questions: 

The evaluation study of services to infants and toddlers included the following 
questions: 

(1) What are the various service delivery models that have evolved? (2) How are 
personnel deployed? (3) In what types of settings and configurations do teachers, 
therapists, and other providers provide services? (4) How are local Infants and 
Toddler programs providing services on a year-round basis and beyond the typical 
working day? (5) Are jurisdictions operating transdisciplinary programs or are they 
deploying providers from various disciplines in sequential formats? (6) What range 
of options do families have within the framework of the Individual Family Service 
Plan? 

Types of Data To Be Collected for Evaluation 

The Maryland SIG will collect the following data: 

Systemic Reform Data: The alignment in policies and procedures and linkages 
across education and other services. Additional data will be collected on proxy measures 
of systemic reform (e.g., development of project services, participation in project services). 
The Maryland SIG will monitor the changes in capacities and practices of schools, 
agencies, and institutions (through the results of technical assistance agreements, annual 
surveys, records of outcomes of LEA grants, and records of cooperation). 

Student Demographic Data: The racial composition of students identified as 
learning disabled, mentally retarded, and as seriously emotionally disturbed. 

Student Outcome Data: Participation of students with disabilities in general 
education, alternative education, and career technology and adult learning. Annual and 
trend data on Maryland Functional Tests, Maryland School Performance Assessment 
Program (MSPAP), LEA norm-referenced performance tests, and optional LEA district- 
wide criterion-referenced tests. Student participation rates, score averages, percent 
passing/failing, by measures, skill tested, grade level, district, and statewide will be 
collected. In addition, Maryland will collect student data on: IEP certificates, diplomas, and 
attendance rates. Maryland will also collect data on suspension and expulsion rates, 
students’ highly challenging school behaviors, and violence and weapons possession. 

Post-School Data: Post-school employment and postsecondary participation rates. 

District Data: The adoption and implementation of new practices, research, and 
grants by Local Education Agencies (LEAs). 
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Preservice Program Implementation, Enrollment, and Follow-up Data: 
Enrollments, graduates, the adoption and implementation of new practices and research, 
and data from graduate follow-up surveys. 

Personnel Preparation Data: Special education trainees and include feedback on 
practitioner’s understandings of new things that are expected under the SIG, practitioners” 
uses of new practices to improve instruction and schoolwide programming, practitioner’s 
needs, and practitioner’s attitudes. 

Feedback Data: From practitioners, consultants, and the Consumer Review 
Groups. 

Family Data: Surveys of family satisfaction with service delivery options in early 
intervention. 

Summative Evaluation Data: The summative evaluation will collect annual data 
on student performance; outcomes of professional development and personnel 
preparation; technical assistance; and research and development information services. 
Summative Student Data will include: average scores of students with disabilities on the 
Maryland Functional Tests and the MSPAP, district criterion-referenced and norm- 
referenced tests. Other data will include the percentage of time in general education by 
students with disabilities, the percentage of students with disabilities receiving high school 
diplomas, participation of students with disabilities in postsecondary education, average 
portfolio ratings on IMPAP, participation in statewide assessments by students with 
disabilities, expulsions and suspensions of students with disabilities, challenging student 
behaviors, placements of students with disabilities in alternative settings, average 
attendance rates of students with disabilities, drop-out rates of students with disabilities, 
and early intervention. 

Types of Analyses To Be Conducted 

Analyses and trend analyses of all data will be completed. Disaggregation of all 
student data will be conducted. In addition, the Maryland SIG will conduct comparative and 
trend analyses of: (a) student data regarding high school diplomas and certificates earned 
by students with disabilities in terms of changes in proportion of students receiving 
certificates, (b) diplomas earned by students with disabilities, (c) students with disabilities 
meeting University of Maryland requirements, (d) the participation of students with 
disabilities in career technology and adult learning career-technology requirements, (e) 
high school program indicators; and (f) proportions of students with and without disabilities 
who achieve outcomes. The Maryland SIG will analyze student employment and 
postsecondary education outcomes. Cross-cutting analyses of data sets on performance 
and outcomes of students with disabilities will also be completed. Trend analysis and 
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comparative analysis of follow-up survey completed by preservice preparation graduates 
will also be conducted. Cross-analyses of profiles of services to infants and toddlers will 
be completed. 

Analyses of Systemic Change: 

The Maryland SIG will evaluate the alignment in policies, procedures, linkages 
across education and other services and data on proxy measures of systemic reform (e.g., 
development of project services, participation in project services). In addition, the SIG will 
monitor changes in capacities and practices of schools, agencies, and institutions 
(particularly through the results of technical assistance agreements, annual surveys, 
records of outcomes of LEA grants, and records of cooperation). 
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New Hampshire State Improvement Grant 

New Hampshire Department of Education Dates: 2/1/99 - 1/31/2003 

Special Education 
101 Pleasant Street 

Concord, NH 03301 Project Director: Robert Wells 

603-271-1536 



Grant’s General Purpose: 

The statewide project in New Hampshire addresses the following system 
components: (1) leadership and service coordination; (2) standards for practice, preservice 
education, and professional development; (3) use of data for short- and long-range 
planning, (4) dissemination of exemplary practices; (5) involvement of individuals with 
disabilities and parents of children and youth with disabilities; and, (6) reduction or 
elimination of systemic barriers to effective service delivery. 

Conceptual Framework for Systems Change: 

The Project goals match the guiding principles of U.S. Office of Special Education 
Programs (OSEP). The grant application described the New Hampshire system, the 
barriers related to professional development, system barriers, and higher education 
barriers. The SIG identified specific systems change strategies from OSEP-funded 
projects. The SIG describes the connections between family participation, research, pre- 
service education, professional development, professional practice, school organization, 
policy, and student outcomes. The application makes references to organizational change 
literature. 

Purposes of the Evaluation 

The purpose is to conduct a comprehensive evaluation of project implementation, 
milestones, and outcomes. 

Evaluation Design and Approach 

The New Hampshire SIG will use formative and summative evaluations to 
evaluate outcomes at three levels. The formative evaluation will focus on the 
implementation of project - project activities conducted as proposed and planned monthly. 
The process evaluation will address project implementation and milestones at 6 months, 
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1 year, and each year. 

The summative evaluation will address intermediate outcomes and the evaluation 
of achievement of objectives by the Project. The summative evaluation of student 
achievement and the impact of project on student learning each year of project will be 
completed. 

The New Hampshire SIG will use a multi-level evaluation of professional 
development based on Guskey’s (1998) and Kirkpatrick’s (1959) frameworks. The 
evaluation of professional development will address participants’ satisfaction, participants’ 
learning, organizational support and change, participants’ use of new knowledge and skills, 
and student learning outcomes. 

Three in-depth case studies of participating districts will be developed. 

Type and Focus of Evaluation Questions 

The New Hampshire SIG has a strong correspondence between goals, objectives, 
and evaluation questions. 

Evaluation questions developed for Goal 1 and Goal 2 and objectives addressed 
the project’s organization, structure, and resources. Most questions require yes/no 
response as to whether activities were conducted. 

Evaluation questions for Goal 3 and Goal 4 and objectives addressed desired 
increases in personnel outcomes in a yes/no format. Evaluation questions for Goal 5 and 
Objectives addressed the establishment of professional development schools, high quality 
retooled programs and activities, and participation of family members or individuals with 
disabilities in activities in a yes/no format. 

Evaluation Questions for Goal 6 and Objectives addressed the involvement of 
families and individuals with disabilities in hiring, in professional development activities, and 
the development and use of standards of family involvement by Local Education Agencies 
(LEAs) receiving project funds in a yes/no format. 

Evaluation questions for goal 7 and Objectives, on the data system, addressed the 
implementation of baseline data, coordination of data systems, and improvement of data 
systems in a yes/no format. 

Evaluation questions for goal 8 and Objectives, on strategies for systemic barriers, 
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addressed in a yes/no format: formation of task forces and ad hoc work groups, enactment 
of new policies and procedures, maintenance of compliance with the Individuals with 
Disabilities Education act (IDEA). 

Evaluation questions for goal 9 and Objectives, on dissemination, used yes/no 
formats to address the development, maintenance and use of a World Wide Web site and 
materials. 

Evaluation questions for goal 1 0, on evaluation, used yes/no formats to address the 
implementation of evaluation studies, data collection and analyses, and the development 
and use of evaluation reports by project. 

Summative evaluation questions include: Were proposed objectives achieved? Did 
achievement of project objectives lead to the desired outcomes in each goal area? Did 
achievement of goals and objectives result in improved performance of children and youth 
with disabilities? 

Types of Data To Be Collected for Evaluation 

Monthly progress evaluation charts will be completed by project staff on 
implementation of activities by goals and objectives. 

Yearly Statewide Student Performance Data: Yearly statewide data will be 
collected on performance of students with disabilities, including graduation rates, drop-out 
rates, and scores on the National Assessment of Educational Progress (NAEP) 
achievement tests. 

Student Performance Data in Participating and Control Local Education 
Agencies (LEAs): Baseline data and yearly outcome data of student performance data 
in 5 participating and 5 control LEAs including graduation rate, drop-out rate, and student 
scores on NHEIAP (statewide assessments). 

Professional Development Data: Qualitative and quantitative measures of 
professional development on numbers and types of professional development activities, 
teachers’ use of practices in classes, and contextual variables. 

Performance indicators: The New Hampshire SIG developed performance 
indicators for: (a) early childhood, (b) elementary and secondary education (including 
academic, postsecondary, personal, social, general indicators of educational success), and 
(c) systemic quality indicators (utilization of standards for quality practice to guide program 
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design and accountability, transition, dissemination and technical assistance). 



Types of Analyses To Be Conducted 



There will be a summative evaluation and analyses of statewide student 
performance, including analyses of trends of statewide student performance data. 

The SIG will conduct an analyses of professional development and student outcome 
data from 5 participating and 5 control Local Education Agencies (LEAs). The SIG will 
complete within site and across site comparisons between study and control LEAs. Three 
in-depth case studies of participating LEAs will be completed within a longitudinal study, 
that includes a comparative analyses of 3 cohorts of student performance data in 
participating districts in comparison to non-participating districts representing rural and 
urban, and small and large districts. 



The SIG will also conduct an analysis of inputs (professional development activities, 
practices used by classroom teachers, contextual variables, and demographic variables); 
and student performance indicators (enrollment in post-secondary education, career status, 
financial well-being, involvement in community activities, and breadth of social networks). 
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Ohio State Improvement Grant 

Ohio Department of Education Dates: 2/1/99 - 1/31/2003 

Division of Special Education 
933 High Street 

Worthington, OH 43085-4087 Project Director: Cynthia Puckett 

614-466-2650 

Grant’s General Purpose: 

The goals of this project include: (1) to provide children and youth with disabilities 
access to challenging academic curricula; (2) to develop partnerships among members of 
the learning community to improve results for all children; (3) to increase cooperation and 
collaboration in and among agencies, and the coordination of services across agencies; 
(4) to promote collaboration between regular and special education personnel to ensure 
successful transitions; (5) to develop a modal to document accountability and continuous 
improvement in service delivery; (6) to improve building-level processes for assisting 
students with learning problems; (7) to increase the participation of parents, families, and 
community members in decision-making; (8) to support the development of shared 
responsibility and ownership at the building level; (9) to make personnel preparation 
programs accessible throughout the state; and (10) to prepare graduates and school 
personnel to effectively manage the behavior of all students. 

Conceptual Framework for Systems Change; 

The Ohio SIG refers to Sashkin and Ergemeier’s (1993) approach: fix the parts, fix 
the people, fix the school, and fix the system. The Ohio SIG describes systems change as 
involving a common vision and direction, the decentralization of authority, and changes in 
accountability. The SIG refers to Ohio’s Transition Systems Change Project findings. The 
SIG identifies Ohio’s model of Continuous Improvement Planning as a systems change 
strategy to gain broad-based stakeholder involvement and consensus. The Ohio SIG will 
identify priority change areas for professional development and related needs and align 
multiple initiatives projects and programs. 

The Ohio SIG describes two major efforts to bridge gaps in Ohio’s educational 
system: implementing a minimum total of 16 local/regional partnership site agreements by 
the 3 rd year; and implementing partnership agreements to address: (a) personnel 
shortages, (b) the evaluation of effectiveness, and (c) the dissemination of information. 

Anticipated systems change outcomes include: (1) improved capacities to direct 
personnel development needs, (2) increased use of external resources to meet 
professional development needs, (3) improved capacity of resource providers to meet 
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needs of consumers, (4) increased use of Ohio’s Continuous Improvement Planning (CIP) 
model to address the needs of students with disabilities, and (5) improvement in state’s 
capacity to meet the needs of children and youth with disabilities. 

Purposes of the Evaluation 

The Ohio SIG identifies two major purposes: 

(1) Monitor progress toward goals and initiatives. 

(2) Evaluate the effectiveness of project activities in improving the state’s 
ability to meet the academic and behavioral needs of children and youth with disabilities. 

Evaluation Design and Approach: 

The evaluation of Ohio’s SIG will be implemented through the State University 
Education Deans (SUED) (representing 14 state Institutions of Higher Education, an 
Evaluation Board, and coordinated by the Evaluation Services Center at University of 
Cincinnati. The Ohio SIG’s Evaluation Board will include representatives from the 16 
regional evaluation centers; therefore a dean from each region will be involved. 

The SIG will use a three-tiered design with process and outcome components: 

Level 1: Local Planning and Accountability - Regional SUED colleges of 
education will serve as evaluation centers for each region and help set local outcome 
objectives related to Continuous Improvement Planning (CIP). The Regional Evaluation 
Centers will monitor progress, collect data, and disaggregate data on statewide 
performance indicators. 

The Coordinating Councils will: (1) identify desired local outcomes and set local 
benchmarks for continuous improvement performance indicators, (2) collect data on 
performance indicators and local outcomes, (3) evaluate the extent to which goals, 
objectives, strategies, and outcomes were achieved, and (4) evaluate the impact of project 
activities on improving academic and behavioral results for children with and without 
disabilities. 

Level 2: Statewide Coordination of Planning & Accountability - The Evaluation 
Services Center at University of Cincinnati will coordinate local evaluation efforts and 
collect and analyze statewide performance indicators. 

Level 3: Evaluation Audits and Self-Studies. An Evaluation Audit and Self-Study 
will be completed by each local partnership site, the Coordinating Councils, and the state 
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team. Key audit components will include the project objectives and progress on 
performance indicators. 

An External Evaluator will conduct site visits and interviews. All regions will attend 
a guided focus group meeting to discuss statewide and local progress and identify areas 
of exemplary progress and needs. The external evaluator will summarize focus group 
findings and recommendations. 

Type and Focus of Evaluation Questions 

The Ohio SIG has a strong correspondence between goals, objectives, and 
evaluation questions. 

Ohio’s SIG Identifies 9 project goals with implementation objectives and evaluation 
measures. The 9 goals address: (a) partnership agreements and coordinating councils, (b) 
the use of a continuous improvement planning process to facilitate training and technical 
assistance, (c) the coordination of professional development and technical assistance, (d) 
the replication of partnership findings throughout Ohio, (e) increased participation of 
parents/families/community members within partnerships, (f) the Superintendents Advisory 
Council serving as project advisory council, (g) identified personnel shortages, and (h) 
evaluation of project. 

Ohio refers to the SIG as results-oriented and that SIG dollars will answer the 
following questions: 

Do we have the necessary number of staff? 

Are they properly trained to meet the needs of today’s and tomorrow’s students? 

Are the answers to these questions the same at the preservice and inservice levels? 

Are more students taking and passing proficiency tests? 

Are more students graduating and completing vocational education programs? 

Are more students leaving high school prepared for higher education? Are more 
students prepared to be productive members of the work force? 
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Types of Data To Be Collected for Evaluation 

Ohio will identify measurable, observable benchmarks for performance indicators. 
The Evaluation Board will collect baseline data on each of the performance indicators in 
the first year. 

Ohio Student Performance Indicators include the: (a) exemption, participation, 
and performance by students with disabilities on statewide and district-wide assessments; 
(b) decreases in exemptions of students with disabilities on statewide proficiency tests; (c) 
increases in graduation and vocational completions by students with disabilities; (d) 
decreased drop out rates by students with disabilities, and (e) increases in students with 
disabilities achieving grade-level benchmarks in subjects. 

Transition and Post-School Student Indicators include: (a) increase in the 
numbers of graduates with disabilities in higher education; (b) increase in the number of 
youth competitively employed; (c) increases in collaboration between general and special 
education and agencies to ensure transition. 

Parent Participation and Satisfaction Indicators include: increases in the level 
of participation and involvement of parents; increased in the number of toddlers 
transitioning to the school-age system; and decrease in complaints and requests for 
impartial due process hearings. Parent/Community Partnership indicators include: 
increases in the level of participation and involvement of parents and community 
representatives; decreases in the complaints and requests for impartial due process 
hearings; and increases in the amount of human and fiscal resources provided to schools. 

Agency Collaboration Indicators include: increases in the numbers of 
partnerships across agencies; increases in the number of infants/toddlers transitioning 
from Part C to preschool; increases in the number of children making successful transitions 
from preschool to primary to middle/junior high schools to high schools, and increases in 
the number of children receiving services external to schools. 

Service Delivery Indicators include: (a) Increases in students graduation, 
employment, and participation in higher education; (b) increases in the use of 
performance-based data for decisions; (c) the development of alternative statewide 
assessments; and (d) increases in the number of schools and districts with local 
accountability systems. 

Building-level Process Indicators include: (a) Increase in the number of student 
objectives on Individualized Education Programs (lEPs) being completed and/or 
benchmarks met; (b) decreases in home instruction for students with serious emotional 
disturbance (SED); (c) reduction in the disproportionate identification of minority, limited- 



OH4 




55 



Ohio Evaluation Profile prepared by RMC Research Corporation, August, 1999 

English-proficient, and economically disadvantaged children with serious emotional 
disturbance (SED) or DH; and (d) decreases in the number of students suspended or 
expelled. 

Personnel Preparation and Training Data will be collected on measurable and/or 
observable outcomes in one or more of the following: teaching practices; student 
outcomes; system policy and procedures; family/community understanding, involvement, 
and support; and administrative practices. 

Regional Local Outcome Data: 16 Regions will develop local outcomes for 
continuous progress plans and collect data on progress toward local outcomes 

Types of Analyses To Be Conducted 

The Ohio SIG will analyze data related to performance indicators and disaggregate 
data on statewide performance indicators to the local region. 

Analyses of Systemic Change 

The Ohio SIG anticipates that systems change outcomes will include: (1) improved 
capacities to direct personnel development needs, (2) increased use of external resources 
to meet professional development needs, (3) improved capacity of resource providers to 
meet needs of consumers, (4) increased use of Ohio’s Continuous Improvement Planning 
(CIP) model to address the needs of students with disabilities, and (5) improvement in the 
state’s capacity to meet the needs of children and youth with disabilities. 
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Utah State Improvement Grant 



Utah State Office of Education 
Services for At Risk Students 
250 East 500 South 
Salt Lake City, UT 841 1 1 

Grant’s General Purpose: 



Dates: 1/1/99-12/30/2003 

Project Director: Bruce Schroeder 
801-538-7711 



To implement the State Improvement Plan which in turn is aligned with the Utah 
State Plan and the Utah Agenda. Three over-riding goals drive the SIG: 



(1) A quality decision-making infrastructure: The state will create a “decision- 
making infrastructure” to improve every aspect of the service delivery 
system. 

(2) The general curriculum: the state will implement changes in policy and 
personnel preparation so that students with disabilities have meaningful 
access to the general curriculum. 

(3) Proactive behavioral/social interventions. The state will provide technical 
assistance activities on proactive behavioral intervention strategies designed 
to assist students with disabilities in achieving he attitudes and competencies 
necessary to be “successful, caring, members of society.” 



Conceptual Framework for Systems Change: 

Based on principles of systemic change and the Utah Agenda for Meeting the 
Needs of Students with Disabilities (strategic plan for the future of students with 
disabilities). Principles of systemic change on which the SIG is based: 



1. Articulating a clear vision for systems change based on identified needs and 
priorities. 

2. Systems change is most effective when it is simultaneously acted upon in 
multiple parts of the organization. 

3. Strong leadership at every organizational level. 

4. Systems change focuses on changing behavior, then attitudinal change will 
follow. 

5. Systems change is most effective when there is organization-wide accountability 
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These principles guide both the SIG and the Utah Agenda. The Utah Agenda articulates 
a number of systemic change strategies. These strategies include developing 
comprehensive and unified systems for funding, personnel development, and services to 
children and youth with disabilities. The Agenda promotes a unified advocacy approach 
to all aspects of service delivery. 

Purposes of the Evaluation 

A twofold purpose is given: (1) To assess the efficiency and effectiveness of the SIG 
design and content in meeting the proposed goals and objectives; and (2) to examine the 
impact of SIG systems change model on Utah’s professional development needs 

Evaluation Design and Approaches 

A management-by-objectives approach will be employed to assess progress 
toward SIG goals and objectives. The objective under each of the three goals is linked to 
outcome measures and personnel responsible. Utah has also recruited national experts 
for each of the three goals to be part of a “National Experts Panel." The Panel is involved 
with the evaluation of the project. 

Type and Focus of Evaluation Questions 

Evaluation questions have been developed for each objective. Overriding question 
pertains to impact of SIG relative to three goals. Examples of specific questions are: 

(1) Is there an increase in the number of students with disabilities taking part in 
statewide and local assessments? (2) Is accountability assessment information being used 
to improve instructional practices for students with disabilities and increase their 
participation in general education? To prevent and treat reading failure? (3) Have 
accreditation processes been aligned with State Improvement Plan priorities? (4) Have 
shortages in high incidence/mild moderate certification been addressed? In rural/remote 
areas? In speech-language pathologists? Birth-to-five certification? 

Types of Data To Be Collected for Evaluation: 

A variety of types of data are to be collected. These include quantitative and 
qualitative data and a variety of outcome data. Some specific types of data to be collected: 

Student Outcome Data: Student performance on state achievement testing and 
Stanford Achievement Test results; and measures of “Child Well Being.” 
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Post-school Outcome Data: One-year follow-up studies of students exiting special 
education. 

District Data: Data on SIG activities in Local Education Agencies (LEAs_ 
participating and not participating. 

Personnel Certification Data: Teaching Field Index that measures degree of 
personnel shortages. Number of Personnel on Emergency Certification 

Paraprofessional Personnel Needs 

Consumer Evaluation of Special Education Services, 

Types of Analyses of Data to be Conducted 

The types of analytic techniques are not described in detail. The evaluation of SIG 
activities will include a comparison of districts targeted for SIG activities and those districts 
not participating directly in the SIG. 

Analyses of Systemic Change 

The analysis of systemic change is infused into the Utah accountability system as 
well as the evaluation components of the SIG. One strategy of the Utah Agenda is “ to 
infuse the Utah Agenda into the Utah Public Education Strategic Plan and other broad 
system change initiatives.” This strategy appears to guide the SIG evaluation process. 
Therefore, while there is no separate evaluation of systemic change, an evaluation of the 
SIG goals and objectives are intended to provide overall measures of changes in the 
service delivery system. 
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Virginia State Improvement Grant 



Dates: 2/1/99 - 3/31/2003 



Project Director: 
804-225-2071 



Karen Trump 



Virginia Office of Education 
Office of Special Education and 
Student Services 
P.O. Box 2120 
Richmond, VA 23214-2120 

Grant’s General Purpose: 
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Conceptual Framework for Systems Change: 
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Purposes of the Evaluation. 

The purposes of the evaluation within the Virginia SIG are as follows: 

(1) Continuously inform leadership and relevant stakeholders of the status of the 
project 

(2) Continuous improvement of the project 

(3) Communicate the value of the project to relevant stakeholders 
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(4)Evaluate achievement of performance goals and objectives yearly and at the end 

of five years 

Evaluation Design and Approaches: 

The Virginia SIG describes a Learning Orientation for the evaluation which will 
address factors that influence emerging and final results and unintended effects. The 
evaluation of the project’s implementation will address results-based program 
improvement, partnership infrastructure, personnel, and a coordinated system of technical 
assistance and professional development The SIG will evaluate results, activities, inputs, 
and contextual factors for continuous improvement. 

The SIG will also use a Customer Orientation to evaluate the quality of the project’s 
activities/services as assessed by school divisions, professionals, parents, IHE faculty, and 
other stakeholders. 

A Results Orientation will be used to evaluate the achievement of planned results 
and unintended effects. The SIG will conduct an evaluation of Results-Based Program 
Improvement on the training and application of knowledge and skills. The SIG will also 
evaluate “sliver grants” to Local Education Agencies (LEAs) and identify successful 
strategies. 

A Strategic Evaluation will be conducted yearly and at the end of five years on the 
accomplishment of performance goals and indicators for the SIG’s three strategic 
directions. 

Types and Focus of Evaluation Questions: 

The Virginia SIG includes a variety of evaluation questions. 

Questions with Results Orientation - Is the planned result being achieved? Was 
the planned result achieved? What unintended effects are occurring or occurred? 

Questions with Learning Orientation- What factors in the project’s actions are 
influencing emerging results and in what ways (formative causal inferences)? What factors 
in the project’s actions are influencing final results and in what ways (summative causal 
inferences)? What unintended effects are occurring or occurred? 
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Evaluation Questions on the Effectiveness of Implementation 

1. Results-based program improvement. Have training institutes been delivered 
to all local administrators and special educators? What were the results? 

2. Partnership infrastructure Have state-level partners been identified and formal 
letters of agreement signed? Do partners meet on a regular basis? Do partners and 
leadership report ongoing satisfaction with the infrastructure? 

3. Focus on persons who work with students with disabilities, including 
families. Has training and technical assistance been provided by Parent Educational 
Advocacy and Training Center (PEATC)? What are the results? Was the World Wide Web 
site developed with what utility? 

4. Coordinate a statewide system of technical assistance and professional 
development. Has the professional development activities coordination group been 
formed as designed? What is the impact of cultural competence training? Have the 
IHE/LEA partnership training grants resulted in improved collaboration skills for general 
and special educators, related services personnel, and administrators? 

5.Support efforts to increase supply and demand of qualified personnel. Has 
the subgrant been awarded for a VA IHE to subpartner with University of Louisville? Are 
all proposed training initiatives delivered as designed? What is the diversity of personnel 
trained through training initiatives? 

Strategic Evaluation Question - At the completion of 5 years, have five year 
performance goals been achieved? 

(The Virginia SIG will assess achievement yearly on the three strategic directions) 
Identified performance goals undereach strategic direction (e.g., Increase 5% each year 
the percentage of students completing school from 66% to 75%) 

Consumer evaluation on quality of activities and services - (1) From 
perspectives of stakeholders what’s working and what’s not? (2) From perspectives of 
stakeholders what are the emerging trends and issues that could influence, positively or 
negatively, the successes of the project? 

Feedback - (1) Do all potential uses of evaluation information have timely access? 
(2) Are all relevant sources of evaluation information available from partner programs for 
decision makers? 
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Types of Data To Be Collected for Evaluation 

Key Contextual Factors will be identified. 

Partnership data: Letters of agreement, meeting records, interviews. 

Personnel & Licensure Data: the number of qualified and not qualified personnel 
employed/contracted; the number of qualified and not qualified personnel 
employed/contracted retained; the number of interstate licensure agreements. 

Personnel Preparation Data: the number and location of approved programs 
statewide; the number and type of interstate agreements; the number of graduates 
completing approved programs; the number of students from traditionally under- 
represented populations prepared. The SIG may include the number of paraprofessional 
training programs developed and the number of people prepared. The numbers and types 
of workshops/inservices offered by the SEA, LEAs, Regional Training and Technical 
Assistance Centers, and the Best Practice Centers. The numbers and types of participants 
and the number of general education teachers and other professionals who participated 
as a component for license renewal. Training evaluations will be collected at the end of 
each training and at least 1 month after training. Data on training by the Parent Educational 
Advocacy Training Center will include data on workshop attendees, processes, and results. 
Cultural competence training data on attendees, interventions, collected at end of each 
training event and at least 2 months after on impact on practice. Data on partnership 
between a Virginia I H E/University of Louisville will be collected. Data on the delivery and 
design of all training initiatives will be collected. Data on diversity of personnel training 
through training initiatives will be collected. Data on short and intermediate results of each 
training initiative will be collected. 

Statewide system of technical assistance and professional development data 

will include: records of meetings and quarterly satisfaction interviews with Professional 
Development Activities Coordination Group. 

World Wide Web data: site usage data and random customer feedback surveys. 

Preschool and early intervention efforts on numbers of children identified, 
transitioned, not eligible under Part B, and numbers no longer in need of special education 
services. 

Family Participation Data: percentages of families participating in eligibility, 
Individualized Education Program (IEP) development, Individualized Family Services Plan 
(IFSP) development, transition from early intervention to preschool, and transition planning 
for students age 14 and older. Parent participation on local advisory committees. Numbers 
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of workshops for parents and numbers of parents trained. 

Student Participation Data: Numbers and percentages of students with disabilities 
enrolled in vocational education courses, percentages of students with disabilities who 
receive all of their Special education or related services in a special education setting. 
Comparison of Least Restrictive Environment (LRE) settings for children ages 3-5 with 
disabilities and for infants and toddlers. Percentages of students with disabilities in grades 
3,5,8, and high school whose IEP documents no participation in assessments. 

School Division and Student Performance Data: Annual collection of data from 
all school divisions on each indicator for the 3 strategic directions: percentages of students 
with disabilities completing school each year; percentages of students with disabilities, 
ages 14-21+ dropping out each year. Percentages of students with disabilities who drop 
out in comparison to students who stay in school in grades 7-12 each year. Percentages 
of students with disabilities who complete school and receive advanced or standard 
diploma, special diploma, certificate of completion, or GED, or reach maximum age of 
eligibility. Percentages of students with disabilities receiving passing test scores on 
Standards of Learning. Performance of students with disabilities on alternate assessments. 
Percentages of students with and without disabilities with long-term suspensions and 
expulsions. Rates of truancy reduction for participating students in truancy reduction 
programs. 

TYPES OF ANALYSES OF DATA TO BE CONDUCTED: 

The Virginia SIG will conduct the following analyses of data: analysis of partnership 
data for satisfaction indicators. Student performance data will be disaggregated and 
reported by disability, age, gender, and race/ethnicity. Student drop-out prevention data 
will be disaggregated from Project Yes. The Virginia SIG will conduct an Integrated 
analyses of performance indicators for school divisions. Aggregation of observations of 
scanning teams and reports will be completed. Aggregation of school division data on 
performance indicators will be completed. Virginia SIG will analyze trends on performance 
indicators that correlate with school completion, professional development, and 
parent/student involvement. 

ANALYSES OF SYSTEMS CHANGE: 

Strategic Evaluation: Progress toward each performance goal will be evaluated 
annually. At the completion of 5 years, the Virginia SIG will evaluate if the five year 
performance goals have been achieved? The Virginia SIG also identified performance 
goals under each strategic direction (e.g., Increase by 5% each year the percentage of 
students with disabilities completing school). 
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